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ABSTRACT 

This  report  provides  a  summary  of  Virtual  Battle  Experiment- AS4  held  at  DSTO-Edinburgh  in 
August  2003  in  support  of  TTCP  MAR  TP-1  (Command,  Control  and  Information  Management). 
The  study  was  designed  to  investigate  possible  picture  compilation  benefits  provided  by  a 
maritime  tactical  network.  Novel  human-in-the-loop  simulation  methodology  was  employed  in 
which  a  networked  virtual  submarine  undertook  a  surveillance  patrol  with  track  information  fed 
to  three  separate  Track  Managers.  Data  analyses  compared  resulting  picture  quality  and  human 
performance.  The  findings  of  the  study  were  that  networking  assisted  picture  completeness  and 
detection  continuity  at  the  cost  of  higher  operator  workload 
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Virtual  Battle  Experiment  AS-4:  Explorations  of 
Networked  Maritime  Operations 


Executive  Summary 

This  document  presents  an  overview  of  a  study  entitled  Virtual  Battle  Experiment- 
Australia  4  (VBE  AS-4).  The  study  was  conducted  in  August  2003.  VBE  is  the  name  given 
to  a  series  of  such  studies  being  carried  out  as  part  of  a  collaborative  programme  within 
The  Technical  Cooperation  Program  (TTCP)  Maritime  (MAR)  Technical  Panel  (TP)  1 
(Maritime  Command,  Control  and  Information  Management). 

The  purpose  of  the  VBE  series  has  been  to  investigate  potential  picture  compilation 
benefits  provided  by  networking  within  the  maritime  domain.  VBEs  involve  immersion  of 
human  participants  within  a  realistic  synthetic  environment,  providing  them  with  a  suite 
of  tactical  applications  and  observing  picture  compilation  in  detail.  The  VBE  concept 
required  that  this  simulation  capability  be  provided  by  a  common  environment,  which 
could  be  integrated  with  national  applications  and  systems  in  any  country.  It  was  also 
required  that  the  simulation  have  the  flexibility  to  allow  individual  countries  to  develop 
and  integrate  their  own  models.  The  Virtual  Maritime  Systems  Architecture  (VMSA), 
developed  by  DSTO,  was  chosen  to  support  this  environment.  The  VMSA  also  supports 
software  such  as  ownship  helm  control  to  allow  the  virtual  platforms  hosting  these 
applications  to  dynamically  interact  with  the  synthetic  environment.  High  level  objectives 
for  the  sequence  of  VBEs  are  many.  They  include: 

1.  Provide  a  framework  to  quantify  the  impact  of  a  Network  Enabled  Capability. 

2.  Examine  types  of  information  to  be  exchanged  between  coalition  partners. 

3.  Measure  the  picture  compilation  benefits. 

4.  Develop  and  assess  different  concepts  related  to  picture  compilation  within  a 
Network  Enabled  Capability.  These  concepts  can  cover  anything  from 
methodologies  for  Track  Number  allocation  through  to  command  team 
interactions. 

5.  Investigate  human  factors  issues  related  to  a  Network  Enabled  Capability. 

6.  Demonstrate  these  benefits  and  concepts  to  the  stakeholder  community. 

7.  De-risk  Network  Centric  Warfare  (NCW)  related  technology  prior  to 
transitioning  to  sea  trials. 

8.  Help  direct  future  research  and  identify  exploitation  routes  through  to  the  fleet. 

The  first  VBE  (VBE- A)  was  held  in  the  UK  in  May  2002:  primarily  an  integration  exercise 
to  demonstrate  that  the  physical  architecture  proposed  for  VBEs  was  appropriate.  Since 
VBE-A  a  number  of  studies  have  been  held  in  support  of  both  collaborative 
experimentation  within  TTCP  and  individual  research  programmes.  As  such,  an  objective 
of  the  present  experiment  was  the  further  development  and  demonstration  of  the 
infrastructure  and  experimental  conduct  required  to  facilitate  future  VBEs  in  support  of 
individual  national  programmes.  Secondary  objectives  of  VBE  AS-4  were  to  examine  three 


simple  NCW  related  hypotheses.  These,  introduced  later,  provide  the  major  focus  of  this 
report.  A  novel  concept  that  has  been  termed  'Concurrent  Comparison'  was  developed  to 
allow  us  to  comment  on  the  hypotheses  examined.  This  method  has  not  yet  been 
developed  to  the  extent  that  would  enable  inferential  hypothesis  testing  however. 
Therefore  the  present  study  should  be  considered  exploratory. 

The  scenario  used  for  the  study  involved  four  coalition  platforms.  Ownship  (a  virtual 
submarine)  was  required  to  undertake  covert  surveillance  of  an  area  where  hostile  vessels 
were  expected  to  be  transiting  in  addition  to  local  merchant  and  fishing  traffic.  The  other 
coalition  platforms  were  two  frigates  and  an  unmanned  air  vehicle  which  were  all  able  to 
transmit  tactical  track  data  to  ownship.  The  scenario  was  entirely  scripted  with  the 
exception  of  the  virtual  submarine. 

In  terms  of  picture  quality,  VBE  AS-4  supported  the  view  that  sharing  of  track  information 
between  coalition  partners  allows  individual  platforms  to  maintain  a  more  complete 
representation  of  the  environment.  However  it  was  not  possible  to  explore  whether  this 
more  complete  picture  provided  any  benefit  in  achieving  the  overall  operational  objective. 
This  is  likely  to  be  an  ongoing  issue  for  some  time  with  VBEs,  in  that  it  is  perfectly  feasible 
to  measure  improvements  in  the  overall  tactical  picture  provided  by  networking,  but  how 
this  relates  to  overall  mission  effectiveness  is  considerably  more  difficult  to  ascertain. 
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1.  Introduction 


1.1  Overview 

This  document  presents  a  study  of  the  impact  of  networking  a  conventional  submarine  with 
vessels  within  a  coalition  vessel  grouping.  The  study  was  entitled  Virtual  Battle  Experiment 
(VBE)  Australia  (AS)-4.  It  was  conducted  August  5,  2003,  DSTO  Edinburgh,  Australia.  The 
name  VBE  refers  to  a  series  of  experiments  being  undertaken  both  within  Maritime 
Operations  Division  (MOD)  DSTO,  and  The  Technical  Cooperation  Programme  Maritime 
Technical  Panel  -1  (Maritime  Command,  Control  and  Information  Management).  The  purpose 
of  these  experiments  is  to  investigate  and  quantify  the  possible  benefits  to  be  gained  by  the 
sharing  of  tactical  information  via  coalition  data  networks,  that  is,  by  implementing  network 
enabled  maritime  operations  of  the  type  likely  to  emerge  in  an  era  of  Network  Centric 
Warfare  (NCW).  A  detailed  discussion  of  the  utility  of  the  NCW  concept  is  beyond  the  scope 
of  this  paper.  The  paper  addresses  a  specific  implementation  of  the  general  NCW  concept  that 
involves  sharing  of  tactical  track  level  data  in  real-time  between  Naval  task  group  elements. 

Within  VBEs,  human  participants  are  immersed  in,  and  interact  with,  a  computer-generated 
environment.  They  are  provided  with  a  suite  of  tactical  and  track  management  applications 
and  are  able  to  manoeuvre  their  virtual  platform  through  computer-generated  water  space. 
The  scope  of  VBEs  can  be  targeted  to  specific  aspects  of  combat  system  functionality  or 
broader  tactical  issues.  It  is  intended  that  the  VBE  concept  will  allow  assessment  of  individual 
components  or  the  capture  of  complex  interactions  that  occur  onboard  real  platforms  [1,  2]. 

A  Virtual  Maritime  Systems  Architecture  (VMSA)  [3],  developed  by  DSTO,  provides  the 
synthetic  environment.  Software  such  as  ownship  helm  control  can  also  be  provided  to  allow 
the  virtual  platform  hosting  these  applications  to  interact  with  the  computer-generated 
environment.  Virtual  platforms  are  provided  with  a  sensor  suite  to  simulate  interaction  with 
real  world  objects. 

The  VBEs  focus  on  the  applications,  algorithms  and  information  exchange  requirements  that 
might  support  picture  compilation  within  a  network  enabled  coalition  as  a  simple 
approximation  of  an  NCW  relevant  operation  -  rather  than  the  physical  communication 
infrastructure  per  se. 

High  level  objectives  for  the  programme  of  VBEs  included  [1,  2]: 

•  Provide  a  framework  to  quantify  the  impact  of  a  network  capability. 

•  Examine  types  of  information  to  be  exchanged  between  coalition  partners. 

•  Measure  the  picture  compilation  benefits. 

•  Develop  and  assess  different  concepts  related  to  picture  compilation  within  a 
Network  Enabled  operation.  These  concepts  can  cover  anything  from 
methodologies  for  Track  ID  allocation  through  to  command  team  interactions. 

•  Investigate  human  factors  issues  related  to  NCW. 

•  Demonstrate  these  concepts  to  the  stakeholder  community. 

•  Help  direct  future  research  and  identify  exploitation  routes  through  to  the  fleet. 
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These  last  two  points  are  particularly  important.  Although  the  transitioning  of  research 
output  through  to  equipment  procurement  programmes  is  rarely  straightforward,  the 
complex  domain  of  NCW  can  make  this  process  especially  difficult.  This  is  primarily  because 
of  the  multifaceted  interactions  between  the  various  components  that  can  contribute  towards 
effective  network  enabling  of  combat  systems  (and  likewise  its  failure).  For  example,  it  is  not 
easy  to  quantify  or  demonstrate  in  isolation  the  overall  operational  benefit  of  a  particular 
application  designed  to  support  NCW.  However  it  is  believed  that  VBEs  have  the  potential  to 
assist  greatly  in  minimising  the  risks  that  may  arise  in  attempting  to  introduce  technologies 
into  Network  Enabled  (NE)  operations.  Experimentation  such  as  these  VBEs  can  make 
important  contributions  at  all  stages  of  the  acquisition  cycle. 

A  summary  of  the  VBEs  that  have  been  held  to  date  is  provided  in  Table  1.  The  first  VBE 
(VBE-A)  was  held  in  the  UK  in  May  2002.  This  VBE  was  primarily  an  integration  exercise  to 
demonstrate  that  the  physical  architecture  proposed  for  VBEs  was  appropriate.  Since  VBE-A  a 
number  of  VBEs  have  been  held  in  support  of  both  collaborative  experimentation  within 
TTCP  and  individual  countries'  research  programmes. 


Table  1  VBE 1  Studies  to  Date 


VBE 

Date 

Location 

Scenario 

Principal  Objectives 

VBE-A 

May 

2002 

UK 

2  platform 
coalition  (ASW) 

VMSA  connectivity  verification. 

VBE-AS1 

Oct  2002 

AS 

2  platform 
coalition  (ASW) 

Develop  baseline  methodology  and 
infrastructure  for  VBEs. 

VBE-AS2 

Dec  2002 

AS 

2  platform 
coalition  (ASW) 

Introduction  of  non-scripted  ownship. 
Development  and  use  of  metrics  for 
detailed  analysis. 

VBE-AS3 

May 

2003 

AS 

4  platform 

coalition 

(ASuW) 

Define  infrastructure  baseline  for  VBE-B. 

VBE-B 

May 

2003 

NZ 

4  platform 

coalition 

(ASuW) 

First  full  five  nation  VBE. 

VBE  AS-4 

August 

2003 

AS 

4  platform 

coalition 

(ASuW) 

Define  performance  baseline  for  future 
VBEs. 

1.2  Objectives  of  VBE  AS-4 

An  overarching  objective  of  VBE  AS-4  was  to  continue  development  and  demonstration  of  the 
infrastructure  and  experimental  conduct  required  to  facilitate  future  VBEs  in  support  of 


1  The  naming  convention  for  VBEs  is  that  those  studies  held  by  an  individual  nation  are  identified  by  an  appropriate  two 
letter  country  code  followed  by  an  incrementing  number.  An  incrementing  letter  identifies  those  held  as  a  collaborative 
experiment. 
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individual  national  programmes.  More  specifically,  three  hypotheses  exploring  NCW  were 
identified  for  investigation.  These  were: 

1.  IF  track  sharing  occurs  THEN  a  more  complete  and  accurate  representation  of  the 
operating  environment  can  be  maintained  by  each  platform. 

2.  IF  track  sharing  of  high  priority  targets  occurs  THEN  they  can  be  more  continuously 
monitored  with  a  greater  accuracy. 

3.  IF  shared  tracks  have  a  realistic  Area  of  Uncertainty  THEN  command  will  have  a 
greater  trust  in  the  information  and  will  associate  and  fuse  them  more  readily  with  own 
ship  tracks. 

These  hypotheses  address  the  most  basic  characteristics  of  NCW  operations,  that  is,  that  in 
receiving  sensor  data  or  tracks  from  coalition  vessels,  networked  vessels  will  have  access  to  an 
expanded  field  of  view  of  the  tactical  situation.  No  one  experiment  will  prove  or  disprove  the 
utility  of  NCW  concepts,  but  testing  hypotheses  such  as  these  explores  practical  issues  that 
will  need  to  be  addressed.  In  addition  to  these  specific  points  a  number  of  other  technical, 
human  and  operational  factors  were  observed.  These  observations  were  intended  to  gain  a 
general  insight  into  aspects  of  NCW  operations  to  be  researched  in  future. 


2.  VBE  AS-4  Summary 

2.1  Scenario  Overview 

The  scenario  for  VBE  AS-4  was  developed  to  provide  a  realistic  low-tempo  setting  (see  Figure 
!)• 

For  VBE  AS-4  (and  all  previous  VBEs  to  date)  the  movement  of  individual  platforms  was 
entirely  scripted  with  the  exception  of  ownship  -  a  simulated  submarine  (called  HMAS 
vWaller).  HMAS  vWaller  had  its  initial  course,  speed  and  depth  pre-defined,  after  which,  the 
Commanding  Officer  (CO)  was  responsible  for  its  manoeuvres.  The  scenario  was  designed  to 
last  up  to  five  hours.  vWaller  was  crewed  by  a  team  of  control  room  operators  performing 
roles  typical  at  sea.  These  roles  will  be  discussed  in  later  sections. 

The  scripted  scenario  was  generated  using  the  Scenario  Generator  developed  by  Defence 
Research  and  Development  Canada,  which  creates  a  script  file  that  can  be  read  by  the 
gameboard  controller  within  VMS). 

2.2  Scenario  Background 

The  following  context  was  provided  to  the  control  room  team  of  vWaller: 

"Intelligence  indicates  that  a  country  to  the  north  of  a  coalition  partner  has  been  steadily 
increasing  the  size  of  its  armed  forces,  with  the  intention  of  expanding  its  borders  to 
incorporate  the  northern  area  of  the  coalition.  While  hostilities  have  not  been  declared  the 
Coalition  Governments  wish  to  prepare  for  a  pre-emptive  strike  in  case  the  situation 
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deteriorates  to  an  unacceptable  level.  This  possible  strike  will  be  in  the  form  of  an  amphibious 
landing.  However  there  is  little  knowledge  of  the  area,  and  how  well  fortified  the  area  might 
be.  Therefore  HMAS  vWaller  has  been  tasked  to  conduct  surveillance  and  reconnaissance, 
prior  to  the  arrival  of  the  task  group. 

The  opposition  controls  sea  and  air  space  out  to  a  range  of  40  NM  from  his  coastline.  HMAS 
v Waller  is  to  conduct  a  covert  patrol,  within  the  40  NM  zone,  and  establish  the  tactical 
environment  prior  to  the  task  force  arriving.  Of  particular  interest  is  any  movement  of 
military  shipping  within  the  local  area.  There  are  two  coalition  vessels  operating  in  the  area. 
The  first  (FFH1)  is  approximately  80  NM  to  the  northeast  and  is  preparing  to  deploy  an 
Unmanned  Airborne  Vehicle  (UAV)  to  search  the  area  to  the  south  east  of  HMAS  vWaller. 
The  second  (FFH2)  is  operating  approximately  80  NM  to  the  northwest  and  is  conducting  a 
surface  search.  North  of  the  current  position  of  HMAS  vWaller  there  is  a  shipping  lane 
running  northwest  to  southeast,  and  to  the  south  west  there  are  fishing  grounds." 


Figure  1  VBE  ASA  Scenario  Overview 

2.3  Sensor  Fits 

The  sensor  fits  for  the  various  platforms,  along  with  their  nominal  detection  ranges  are 
summarised  in  Table  2. 
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Table  2  VBE  AS-4  sensor  fit 


Platform 

Sensor 

Detection  range 

(nautical  miles) 

Detection  Range 

(km) 

HMAS  vWaller 

Sonar 

10 

18.5 

ESM 

20 

37.0 

HMAS  vARUNTA  (FFH  1) 

Radar 

60 

111.0 

HMAS  vANZAC  (FFH2) 

Radar 

60 

111.0 

UAV 

Radar 

30 

55.5 

EO 

10 

18.5 

2.4  Coalition  Information  Exchange 

All  radar  tracks  held  by  the  two  coalition  frigates  were  communicated  to  ownship:  vWaller. 
These  tracks  were  assigned  a  classification  of  air  /  surface  entity  with  an  unknown  hostility. 

All  radar  tracks  and  visual  reports  from  the  UAV  were  communicated  to  vWaller.  If  only  a 
radar  track  was  held  then  this  was  assigned  a  classification  of  air  /  surface  entity  with  an 
unknown  hostility.  Once  the  contact  had  been  detected  visually  the  known  hostility  was 
assigned  to  both  the  visual  and  radar  tracks. 

2.5  HMAS  vWaller  Tasking 

HMAS  vWaller  was  tasked  to  conduct  a  surveillance  patrol  within  40  NM  of  the  coastline  and 
monitor  for  Indications  and  Warnings  of  hostile  intent,  prior  to  the  arrival  of  the  Coalition 
Amphibious  Task  Group. 

The  priorities  for  HMAS  vWaller  were: 

1.  Remain  undetected. 

2.  Establish  surface/subsurface/air  activity. 

3.  Locate  and  track  military  shipping. 

The  Rules  of  Engagement  were: 

1.  You  may  engage  the  enemy  only  in  self-defence. 

2.  You  are  not  to  intentionally  close  any  unit  to  within  8  kyds  as  this  may  be  viewed  as  a 
hostile  act. 

2.6  Picture  Compilation  Task 

A  major  role  of  the  submarine  "Command  Team"  (stationed  in  the  control  room)  is  to  build  an 
accurate  picture  of  the  world  around  them.  This  is  referred  to  as  picture  compilation.  Ground 
Truth  tracks  stored  during  the  scenario  provide  a  comparison  against  which  the  tactical 
picture  compiled  by  human  operators  and  or  automated  tools,  during  the  VBE,  can  be  directly 
compared.  A  simplified  activity  diagram  of  current  picture  compilation  activities,  as  observed 
at  sea,  is  shown  in  Figure  2. 
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The  current  estimated  range,  course  and  speed  of  a  contact  is  referred  to  as  its  solution. 
Situational  or  characteristic  clues  provided  from  sonar,  visual,  radar  or  electronic  support 
measures  (ESM)  constrains  the  uncertainty  surrounding  the  achievement  of  a  solution.  Track 
Management  (TM)  and  Target  Motion  Analysis  (TMA)  is  an  iterative  process  that  is 
conducted  on  every  contact  to  refine  a  Range,  Course  and  Speed  solution  (addressed  in  detail 
later).  Once  a  solution  has  been  assigned  it  is  monitored  constantly  in  reference  to  the  Contact 
Evaluation  Plot  (CEP). 


New  Contact  Initialisation 

eg.  SONAR  Report 
Bearing 
Intensity 

Passive  Ranging 
Speed  Estimates 
Poss  Class 


New  Contact  Initialisation 

eg.  Visual  Report 
Bearing 
Range 
ATB 

Class  /  Poss  Class 
Speed  Estimates 


New  Contact  Initialisation 

eg.  ESM 

Bearing 
Strength  (dB) 

Poss  Class 


New  Contact  Initialisation 

eg.  RADAR 

Bearing 

Range 

Speed 


TM/CEP 

Designates  /  Assigns  Track 
Number  (eg  SONAR  01) 


TM/CEP 

Plots  contact  (eg.  SONAR  01) 


TMA 

Generates  Relative  Motion  Solution  (RMS  - 
►j  Baseline  solution) 

(eg.  Reciprocal  Bearing,  Estimated  speed, 
Estimated  Range) 


Situational  Considerations  eg 

Shipping  Lane  (course) 

Range  of  Day  (max  sensor  distance) 
Proximity  of  Land  (Range) 

Fishing  banks  (Range  /  Course) 
Other  sensor  inputs 


TMA 

Assigns  Solution 
(Bearing,  Range,  Course, 
Speed,  Bearing  rate,  Time) 


1 

TM/CEP 

( 

1  v 

f  A 

Records  solution 

TMA  S 

uuw 

TM/CEP 

(eg.  SONAR  01  U - , 

Reports  adjusted  solution  to 

Accepts 

Reports  to  OOW 

RMS  /  updates) 

OOW  (Bearing,  Range, 

Solution 

(eg.  "New  contact  SONAR  1  + 

J 

Course,  Speed  and  bearing 

1 

Bearing  +  Intensity  +  Possible 

V  J 

rate) 

A - 

Classification) 

J 

C 

t 

TM/CEP 

Monitors  Tracks 


TM/CEP 

Accepts  Tactical 
Picture 


TMA 

Refines  initial  solution  by  adjusting 
Range,  Course  and  Speed 
encoders  to  "stack  the  dots" 


Analysis  Requirement 

Conduct  ranging  manourvre 
Conduct  1936  Ranging 
SONAR 

”  Visual  Updates 
Single  Leg 
RADAR 
ESM  reports 


Figure  2  Basic  Picture  Compilation  Process  [4] 


3.  Infrastructure 


3.1  Overview 

This  section  reviews  the  infrastructure  implemented  for  VBE  AS-4,  and  includes  a  description 
of  the  simulation  environment,  applications,  major  data  flows  between  the  principal 
components  and  display  layout. 

A  high  level  representation  of  the  infrastructure  for  VBE  AS-4  is  shown  in  Figure  3. 
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Figure  3  High  level  view  of  infrastructure  for  VBE  ASH. 

The  actual  VMSA  simulation  environment  modelled  kinematic  data  of  all  platforms  within 
the  scenario  as  well  as  track  data  arising  from  all  coalition  sensors  that  were  present.  Ownship 
C2  applications  within  vWaller  received  data  corresponding  to  its  own  sensors  direct  from  the 
simulation.  However  track  data  from  the  coalition  platforms  was  transmitted  via  dedicated 
TCP  /  IP  links.  Although  this  data  could  be  passed  directly  to  ownship  within  the  simulation, 
the  use  of  an  external  communication  route  is  preferred  within  VBEs  because  it  allows 
different  NCW  communication  methods  and  protocols  to  be  investigated.  It  will  also  more 
readily  support  the  modelling  of  communication  bearers  within  future  VBEs.  A  VBE 
communications  protocol  manipulation  concept  is  under  development  to  simulate  possible 
architectural  characteristics  but  was  not  implemented  in  VBE  AS-4. 

The  Infrastructure  includes  a  set  of  software  models  termed  'federate',  whose  outputs 
simulate  activities  real  world  objects  (RWO)  in  a  simple  tactical  setting  (see  Table  3). 


Table  3  Virtual  Maritime  System  Architecture  Federate  Components 


1.  Visual  Reporting 
Federate 

Simulates  manual  visual  reports  from  a  surveillance  aircraft.  Following  first 
detection  the  "  observer"  waits  until  the  aircraft  has  passed  the  closest  point  of 
approach.  A  fix  is  then  taken  and  track  reported.  After  processing  other  visible 
platforms  are  reported.  For  each  contact,  relative  bearing,  range  along  with  estimates 
of  course  and  speed  are  reported. 

An  Area  Of  Uncertainty,  for  each  contact,  is  defined  as  an  ellipse  whose  major  and 
minor  axes  correspond  to  a  bearing  error  of  +/-  2°  and  a  range  error  of  +/-  25%. 
Estimates  of  elevation,  course  and  speed  are  also  provided  with  errors  of  +/-  6° ,  +/- 
22.5°  and  +/-  25%  respectively. 

2.  Motion  Federate 

A  generic  motion  federate  used  to  model  the  motion  of  simulated  sea-based 
platforms  using  realistic  acceleration,  turn  and  dive  rates. 

3.  U A V/ Surveillance 
Aircraft  Federate 

A  surveillance  aircraft  federate  that  provides  aircraft  kinematic  modelling.  In  the 
future  it  is  hoped  to  link  this  federate  to  a  commercial  flight  simulator  to  provide 
both  dynamic  and  realistic  aircraft  modelling. 

4.  Navigation 

Federate 

A  generic  navigation  federate  used  to  model  the  navigation  capability  of  simulated 
platforms.  For  VBE-B  the  navigation  federate  was  simply  a  replication  of  ground 
truth.  However  the  federate  will  allow  the  impact  of  the  performance  of  the 
navigation  systems  to  be  investigated  in  future  VBEs. 
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3.2  Ownship  C2  Applications 

3.2.1  Horizon  3  Overview: 

Horizon  3  [see  5],  facilitates  management  of  track  data  within  the  vWaller  control  room. 
Central  to  Horizon  3  is  the  use  of  custom  plugins  to  ease  the  introduction  of  third  party 
applications  and  allow  configuration  changes  to  match  varying  user  requirements  (see  Figure 
4). 

Horizon  is  a  developmental  software  tool  under  constant  revision  depending  on  the 
requirements  of  experimentation  and  the  task  of  the  operator.  In  VBE  AS-4,  the  Horizon  3  user 
interface  employed  a  situation  plan  view  (called  a  Tactical  Picture  Display  -  TPD  -  See  Figure 
4a),  together  with  a  set  of  interaction  tools  such  as  zoom,  track  filtering,  annotation,  shoreline 
overlay  and  cursor  position  readouts  (see  [5]  for  details).  The  map  display  also  has  an  'off- 
platform'  mode  to  facilitate  the  manual  cross-fixing  of  bearing  reports  from  sensors  located  in 
different  positions,  including  own-ship  and  coalition  members.  In  addition,  the  operator  was 
able  to  switch  to  a  Time  Bearing  Display  (Figure  4b)  to  observe  the  change  in  relative  bearing 
(bearing  relative  to  ownship  heading).  Further,  Horizon  3  also  provided  an  interface  (see 
Figure  4c)  suitable  for  simulation  of  the  sensor  suite  (sonar,  ESM  and  Periscope). 

3.2.2  Horizon  3  Functionality:  Track  Management 

Track  management  functionality  incorporated  within  Horizon  3  included  manual 
classification  assignment,  track  dropping  and  deletion,  track  "promotion"  to  other  Horizon 
displays,  track  association  and  fusion. 

In  addition  to  allowing  operators  to  select  tracks  for  association  and  fusion  using  an  entirely 
manual  process,  the  release  of  Horizon  3  used  for  VBE  AS-4  incorporated  a  basic  nearest 
neighbour  algorithm.  This  algorithm  provided  recommended  actions,  i.e.  track  association  or 
disassociation  recommendations  according  to  track  data  covariance.  The  algorithm  (see  [5]  for 
details)  is  based  upon  a  "bearings-only"  gating  technique. 
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a.  Typical  Tactical  Display  (TPD) 


b.  Time  Bearing  Display 


c.  "God's  Eye  View"  (Sensor  Controller  Operations) 


Figure  4  Horizon  3  displays  derived  from  VBE  ASH:  Horizon  situation  display 


Operators  were  able  to  fuse  tracks  by  selecting  an  appropriate  fusion  algorithm.  Figure  5 
demonstrates  the  resultant  outcome  of  Track  fusion.  In  Figure  5a  tracks  Ti,  T2  and  T3  are  fused 
to  yield  T4.  Hence,  fusion  involves  "association"  or  grouping  of  two  or  more  component 
"track"  segments  that  are  attributable  to  the  same  RWO  but  from  different  sonar  arrays.  The 
examples  in  Figure  5  refer  to  the  association  of  tracks  derived  from  Flank  and  Towed  arrays. 
Fusion  can  be  handled  in  many  different  ways.  Several  examples  presented  in  Figure  5  are  as 
follows: 

i.  Preferred  Sensor 

For  the  Preferred  Sensor  algorithm  the  operator  was  able  to  assign  a  preferred  priority  order 
for  the  sensors  from  which  associated  tracks  were  derived  (e.g.  Flank  Array  or  Towed  Sonar 
Arrays).  Figure  5b  demonstrates  an  instance  where  Towed  Array  was  selected  as  "Preferred 
Sensor". 
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ii.  Hide  Others 

The  Hide  Others  algorithm  was  similar  to  the  Preferred  Sensor  algorithm;  except  that  only 
track  data  from  the  most  preferred  sensor  was  used.  Note,  however,  that  if  this  track  is 
subsequently  dropped  then  its  history  is  not  updated  until  there  are  more  data  from  this 
sensor.  This  is  shown  in  Figure  5c  where  the  fused  track  T4  follows  the  Towed  Array  tracks, 
Ti  and  T2  ,  and  the  Flank  Array  Track  T3  is  ignored  or  "hidden". 

iii.  Join 

The  Join  algorithm  was  used  to  associate  temporally  disjoint  tracks.  This  algorithm  operates 
so  that,  when  member  tracks  within  a  fused  track  are  concurrent,  fused  track  attributes  do  not 
change  until  one  member  track  is  dropped.  This  is  demonstrated  in  Figure  5d.  Here,  in  a 
similar  manner  to  the  Preferred  sensor  order  algorithm.  Towed  Array  Tracks  Ti  and  T2  and 
effectively  combined  with  the  Flank  Array  track  T3  when  Towed  Array  tracks  are  dropped. 
The  difference  being  that  component  tracks  (Ti,  T2,  T3)  are  temporally  disjoint  (at  ti  and  t2). 
The  association  at  Figure  5d  yields  fused  track  T4  . 


Flank  Array  Sonar  Track 


Fused  Track 


Towed  Array  Sonar  Track 


Preferred  Sensor  Hide  Others 

Association 


Join 


Figure  5  Preferred  Sensor ,  Hide  Others  and  Join  track  fusion  algorithms 


Other  more  complex  algorithms  included  (detailed  discussion  is  beyond  the  scope  of  this 
general  document): 

iv.  Extended  Kalman  Filter 

The  Extended  Kalman  Filter  fusion  algorithm  used  bearing  data  relative  to  the  individual 
sensors  in  order  to  estimate  the  kinematic  characteristics  for  a  fused  track. 
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v.  Closest  to  Source 

The  Closest  to  Source  algorithm  required  at  least  one  of  the  member  tracks  of  a  fused  track 
to  have  a  position  solution.  The  member  track  with  a  position  closest  to  the  appropriate 
sensor  position  is  used  for  the  fused  track  attributes. 

3.3  Target  Motion  Analysis 

Target  Motion  Analysis  is  a  crucial  part  of  the  overall  picture  compilation  and  track 
management  process.  In  VBE  AS-4,  TMA  was  performed  using  the  tool  ITMA,  where  a 
display  of  residual  error  in  a  series  of  range,  course  and  speed  solutions  is  analysed 
graphically  (by  means  of  an  interactive  visualisation).  The  tool  simulates  manual  TMA 
undertaken  at  sea  where  a  series  of  "dots"  (representing  a  sequence  of  solutions  over  time)  are 
"stacked"  in  a  straight  line  by  manipulating  the  range,  course  and  speed  of  a  particular 
contact.  A  set  of  "stacked  dots"  is  achieved  when  the  solution  best  fits  the  geometry  of  a 
contact's  motion  over  time  (range,  course  and  speed  relative  to  ownship  -  see  Figure  6).  ITMA 
is  a  DSTO  developed  display  and  analysis  tool  based  closely  on  equivalent  devices  employed 
at  sea. 


Figure  6  TMA  Display  (ITMA) 

3.4  Coalition  Communication  Mechanism 

3.4.1  Horizon  Promotion  Communication 

Horizon  Promotion  Communication  has  been  used  in  every  VBE  to  date  (barring  VBE-AS1).  It 
is  a  TCP/IP  based  communications  method  adopted  to  enable  physically  distributed  instances 
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of  Horizon  and  other  applications  to  share  track  and  navigational  information.  Within 
Horizon  only  those  tracks  that  are  labelled  for  promotion  are  distributed.  This  process  of 
promotion  can  either  be  manually  or  automatically  initiated. 

All  messages  are  prefixed  by  a  'Message  Header'  that  indicates  why  it  has  been  sent  followed 
by  an  appropriate  number  of  records  each  with  its  own  separate  header  as  shown  in  Figure  7. 


Record  1  Record  2  Record  n 

Message  Header 

Header  Record  Data 

Header  Record  Data 

Header  Record  Data 

Figure  7  Message  and  record  concept  for  Horizon  promotion  communication 
The  possible  message  reasons  that  are  currently  implemented  are: 


•  A  shutdown  request  and  that  the  client  should  not  attempt  to  reconnect. 

•  Initialisation. 

•  A  track  has  been  created  within  the  server's  repository. 

•  A  track  has  been  promoted  from  the  server's  repository. 

•  Track  data  for  a  promoted  track,  other  than  solution  data,  has  been  modified. 

•  Solution  track  data  for  promoted  tracks  has  been  modified. 

•  A  new  source  platform  (i.e.  a  coalition  vessel)  has  been  created. 

•  A  source  platform  has  been  deleted. 

•  Kinematic  source  platform  data  has  been  created. 

•  A  track  has  been  demoted 

•  Track  association  changes 

There  are  currently  six  different  record  types  that  have  been  developed  for  use  with  Horizon 
Promotion  Communications: 


•  Track  details  other  than  solution  related  data 

•  Track  Number 

•  Track  solution  data  (e.g.  bearing,  range,  course  etc.) 

•  Source  platform  details 

•  Source  platform  history 

•  Track  association  changes 

3.5  Summary  Data  Flows  and  Display  Layout 

3.5.1  Data  Flows 

In  VBE  AS-4  the  control  room  of  v Waller  was  simulated  using  a  set  of  displays  that  enabled 
comparison  of  operator  activity.  A  high  level  functional  diagram  of  the  major  ownship  data 
flows  is  shown  in  Figure  8  below.  Four  roles  were  played.  The  roles  wer:  three  track  manager 
roles  plus  a  human-operated  TMA  role  supporting  ownship  track  management  (as  in  current 
practice). 
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Data  flows  and  display  layout  enabled  comparison  of  picture  compilation  undertaken  using 
current  practices  (detection,  localisation  and  tracking  using  ownship  processing)  against  a 
network  enabled  process  (sensor  data  available  from  coalition  vessels).  Note  that  data  from 
coalition  vessels  was  input  to  the  networked  displays  subsequent  to  refinement  of  individual 
track  solutions  performed  by  manual  TMA  upon  ownship  detected  tracks. 


Figure  8  VBE  ASA  Data  Flows 


The  major  roles  identified  above  are  addressed  briefly  in  Table  4  (and  described  in  some  detail 
later). 


Table  4  Operator  roles  in  the  vWaller  control  room 


Title 

Task 

Ownship  Track  Manager 
(OSTM) 

Management  of  tracks  resulting  from  simulation  of  Ownship  sonar  and 
ESM  sensor  detections  at  typical  ranges. 

Network  Track  Manager 
(Net  TM) 

Management  of  tracks  resulting  from  Coalition  sensors  as  well  as  those 
resulting  from  Ownship  sonar  and  ESM  sensors  in  addition  at  typical 
ranges  (supervised  by  CO) 

Trials  Track  Manager 
(Trials  TM) 

Management  of  tracks  resulting  from  Coalition  sensors  as  well  as  those 
resulting  from  Ownship  sonar  and  ESM  sensors  in  addition  at  typical 
ranges  (unsupervised  track  management  -  employing  augmented  track 
management  techniques) 

Target  Motion  Analysis 
(TMA)  -  supports  OSTM  Role 

Utilised  DSTOs  custom  operator  supported  TMA  tool  ITMA,  to  refine  the 
range,  course  and  speed  solution  on  individual  tracks  (employs 
contextual  information). 
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3.5.2  Display  Layout 

In  terms  of  the  Track  Management  as  part  of  Picture  Compilation  activities,  each  operator  was 
supplied  with  two  displays  (see  Figure  9  below).  One  display  was  meant  to  be  used  as  a 
" work  bench"  at  which  tracks  were  identified  and  compared  and  if  possible  fused  to  integrate 
the  tactical  picture.  The  idea  was  that  the  operator  would  promote  those  tracks  that  were 
thought  to  be  valid  associations  (or  valid  unassociated  tracks)  to  a  Tactical  Picture  Display 
(TPD).  Hence  each  Track  Manager  had  an  associated  TPD  display  (as  in  Figure  9). 
Information  on  the  character  of  vessels  being  tracked  (eg.  Classification  by  acoustic  properties, 
best  speed  by  sonar,  visual  contact  and  radar  analysis)  were  supplied  verbally  by  a  sensor 
controller2  (see  4.1.3  for  details  on  this  role). 


Figure  9  Ownship  Display  Layout 

3.6  Location  and  Physical  Environment 

VBE  AS-4  was  undertaken  at  MOD,  DSTO  undersea  battle  lab  (see  Figure  10  below).  Lighting, 
screen  layout  and  screen  luminance  are  set  to  approximate  conditions  experienced  at  sea. 
However,  in  an  effort  to  support  the  CO  situation  awareness,  standard  screen  displays  were 
projected  at  a  large  scale  as  shown  in  the  Figure. 
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Figure  10  MOD's  Under  Sea  Battle  Lab 


4.  Experimental  Conduct 


4.1  Participants 

All  but  one  of  the  operator  positions  played  out  during  VBE  AS-4  involved  experienced  RAN 
personnel.  Although  TMA  was  undertaken  by  civilians  they  were  experienced  with  TMA  and 
the  use  of  the  software.  The  crewing  of  the  vWaller  command  team  is  summarised  within 
Table  5.  Their  roles  are  addressed  below. 

Table  5,  Crewing  for  VBE  AS-4 


Role 

Organisation 

Commanding  Officer 

RAN  Reserve 

Ownship  Track  Manager 

RAN 

Sensor  Controller 

RAN 

Target  Motion  Analysis 

DSTO  (experienced) 

Network  Track  Manager 

RAN 

Trials  TM 

RAN 

4.1.1  Commanding  Officer 

The  CO  had  overall  responsibility  for  vWaller.  For  the  purposes  of  this  experiment  his  two 
principal  tasks  were  the  manoeuvring  of  vWaller  and  the  development  of  the  tactical  picture 
in  accordance  with  the  platform's  tasking.  To  assist  him  in  this  process  he  was  provided  with 
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plan  displays  of  coalition  only  data  (Horizon  -  coalition  console)  and  ownship  only  data 
(Horizon  -  TPD  console). 

4.1.2  Ownship  Track  Manager  (OS  TM) 

The  OS  TM  displays  were  provided  with  track  data  direct  from  vWaller's  onboard  sensor 
models  (Sonar  and  ESM).  These  contacts  were  required  to  be  classified  and  a  solution  refined 
in  the  TMA  process.  The  CO  worked  with  the  ownship  track  manager  who  he  directed  on 
which  tracks  he  would  like  fused  and  when.  The  CO  also  was  able  to  instruct  the  OS  TM  on 
which  tracks  he  would  like  promoted  to  his  command  display  (Horizon  -  command  console). 
This  provided  him  with  an  uncluttered  display  of  the  key  information  contained  within  the 
OS  TM  display  (Horizon  -  track  manager  console). 

4.1.3  Target  Motion  Analysis  (TMA) 

TMA  is  used  by  submarines  to  refine  localisation  of  contacts.  While  it  is  recognised  that  TMA 
at  sea  is  a  broad  concept  involving  several  possible  inputs,  in  this  paper,  TMA  refers  only  to 
the  digitised  processing  of  track  information  with  operator  support.  The  operator  becomes 
directly  involved  in  the  TMA  process  by  editing  the  data  used,  merging  information  from 
different  sensors  (eg.  cylindrical  arrays  and  distributed  array),  applying  constraints  (eg 
sensible  kinematic  course,  or  range  restrictions)  or  using  manual  techniques. 

Manual  TMA  allows  the  operator  to  adjust  the  solution  parameters  and  to  observe  the  effect 
on  the  error  residuals.  This  manual  approach  to  TMA,  whilst  operator  intensive,  is  useful  in 
situations  where  geometry  and  data  quality  can  cause  automatic  TMA  algorithms  to  degrade. 

4.1.4  Network  Track  Manager  (Net  TM) 

The  Net  TM  console  received  track  data  from  ownship  sensors  (sonar  and  ESM)  and  from  the 
coalition  partners.  He  then  assisted  the  CO  in  developing  the  tactical  picture  through 
providing  recommendations  for  fusion  and  undertaking  any  fusion  instructions.  He  also 
assigned  classifications  and  promoted  tracks  to  the  Command  display  as  directed  by  the  CO. 
The  Net  TM  was  permitted  to  provide  guidance  to  the  trials  operator  as  long  as  it  did  not 
impact  on  his  own  tasks. 

4.1.5  Sensor  Controller 

This  operator  was  provided  with  a  Horizon  system  that  provided  a  labelled  plan  display  of 
the  ground  truth  that  was  not  visible  to  the  rest  of  the  v Waller  team.  This  system  also 
displayed  the  tracks  that  were  currently  held  by  ownship  sensors  and  the  RWOs  to  which 
they  corresponded.  Using  this  the  Sensor  Controller  was  able  to  provide  estimates  of  the 
information  that  is  typically  expected  from  sonar,  periscope  and  ESM  departments.  Examples 
of  this  type  of  information  are  range  by  DA  (Distributed  Array),  classification,  and  best  speed 
by  sonar.  The  sensor  coordinator  was  also  responsible  for  operating  the  helm  as  directed  by 
the  CO. 
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4.1.6  Trials  Track  Manager  (Trials  TM) 

The  Trials  TM  was  given  the  freedom  to  act  completely  independently  of  the  CO  and  develop 
his  own  tactical  display  based  upon  ownship  tracks  (sonar  and  ESM)  and  the  tracks  sent  over 
from  coalition.  The  Trials  Manager  was  allowed  to  fuse  any  tracks  of  her  choosing,  when  she 
wanted  and  using  the  algorithms  (see  Fig.  5)  of  her  choice.  This  role  was  introduced  to 
provide  a  comparison  with  the  current  practice  of  the  CO  (or  Officer  of  the  Watch)  having 
responsibility  for  the  fusion  of  external  tracks  with  those  from  ownship.  The  Trials  Manager 
was  also  requested  to  make  recommendations  to  the  Track  Manager. 

4.1.7  Track  Fusion  Procedures 

The  CO,  Net  TM  and  Trials  TM  were  provided  with  the  following  instructions  on  which 
algorithm  to  use  when  fusing  tracks  together. 

1.  If  two  or  more  of  the  tracks  to  be  fused  have  a  position  solution  then  the  preferred  order 
for  the  fusion  algorithms  is: 

i.  Extended  Kalman  Filter  (EKF) 

ii.  Preferred  Sensor 

iii.  Closest  to  source 

2.  If  an  ownship  sonar  or  ESM  bearing  track  is  to  be  fused  with  one  or  more  coalition  tracks 
then  the  preferred  order  for  the  fusion  algorithms  is: 

i.  EKF 

ii.  Preferred  Sensor  (probably  set  to  one  of  the  coalition  tracks) 

iii.  Closest  to  source 

3.  If  ownship  tracks  only  are  to  be  fused  then  the  preferred  order  for  the  fusion  algorithms  is: 

i.  Preferred  Sensor 

ii.  EKF 

4.2  Analysis  Summary:  Series  Comparisons 

The  above  arrangement  of  operators  and  displays  enabled  comparison  of  the  current  process  of 
picture  compilation  to  be  compared  with  future  NCW-like  picture  compilation. 

Current  practice  essentially  provided  a  baseline  against  which  a  NCW  environment  could  be 
compared.  For  this  study  the  baseline  was  the  tactical  picture  compiled  by  vWaller 
independent  of  the  coalition  (where  Track  Manager  extracts  tracks  from  sonar  and  ESM 
detections  and  the  corresponding  TMA  solutions). 

In  summary  current  process  was  represented  by  providing  simulated  organic  sonar  detection 
data  to  a  conventional  Track  Manager  working  together  with  a  Target  Motion  Analyst  to 
compile  the  tactical  picture. 
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NCW  operations  were  represented  by 

Net  TM:  This  operator  was  supervised  by  the  CO  and  was  supplied  the  output  of  the  OS  TM 
(organic  sensors  as  in  current  practice)  together  with  contact  data  supplied  by  coalition  vessels 
(including  a  UAV). 

Trials  TM:  supplied  the  output  of  OS  TM  but  provided  with  a  suite  of  algorithms  and  given 
licence  to  exploit  the  networked  information. 

4.3  Observers 

VBEs  require  a  number  of  formal  observer  roles.  These  are  given  in  Table  6. 


Table  6,  VBE  AS-4  Observers 


1.  VMS  A  Observer 

The  VMSA  observer  ensured  that  the  simulation  was  running  as 
expected.  Any  peculiarities  observed  that  whilst  not  halting  the  conduct 
of  the  experiment  may  impact  on  subsequent  analysis  or  future  VBEs 
were  to  be  reported. 

2.  Infrastructure 
Observer 

The  infrastructure  observer  ensured  that  all  the  applications  and 
supporting  infrastructure  were  operating  as  expected  and  that  all  the 
required  data  flows  were  in  place. 

3.  Conduct  Observer 

The  conduct  observer  monitored  the  overall  conduct  of  the  experiment 
and  identified  key  moments  within  the  experiment  and  important  issues 
associated  with  the  performance  of  the  team  from  a  human  factors 
perspective.  The  conduct  observer  also  maintained  an  informal  log  of 
the  experiment. 

4.  Horizon  Observer 

The  Horizon  observer  monitored  the  overall  performance  of  Horizon, 
and  identified  any  issues  associated  with  the  use  of  this  application  with 
particular  attention  paid  to  its  operability  and  algorithm  performance. 
Aspects  of  the  HCI  that  the  operators  found  difficult  to  use,  or  used  in  a 
different  way  from  that  intended  were  also  noted. 

5.  Photographer 

Responsible  for  creating  a  photographic  and  video  record  before  during 
and  after  the  conduct  of  the  experiment. 

6.  Coordinator 

The  coordinator  had  overall  responsibility  for  ensuring  that  the 
experiment  was  conducted  in  a  manner  as  specified  in  experimental 
plan  or  as  could  best  be  achieved  whilst  still  supporting  the  objectives  of 
the  experiment. 

4.4  Workload  Monitoring  and  Situation  Awareness 

4.4.1  Workload 

Data  overload  is  often  discussed  as  a  possible  risk  for  NCW  capability.  The  human  operator  is 
vulnerable  to  information  flooding.  However,  cognitive  workload  in  particular  is  not  an  easy 
thing  to  measure.  Measures  of  workload  are  typically  survey  instruments  such  as  NASA's 
Task  Load  indeX  (NASA  TLX). 
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Such  surveys  are  most  often  given  after  the  completion  of  a  task.  In  a  wargame  scenario  it  was 
anticipated  that  workload  would  increase  and  decrease  dramatically  so  it  was  decided  to  trial 
a  very  simple  moment-to-moment  technique  developed  at  Space  and  Naval  Warfare  Research 
Centre  (SPAWAR)  San  Diego  [9].  The  technique  simply  requires  the  operator  to  rate  their 
perceived  workload  at  regular  intervals  throughout  the  scenario  using  a  simple  1-7  rating 
scale  implemented  in  a  pop-up  window  each  five  minutes.  This  technique  was  applied  to  all 
operators  in  the  control  room. 

4.4.2  Situation  Awareness 

In  addition  to  the  moment-to-moment  measure,  a  survey  developed  at  Naval  Undersea 
Warfare  Centre,  and  based  on  Endsely's  Situation  Awareness  Rating  Technique  (SART)  [10] 
was  also  delivered  to  the  control  room  participants  at  the  completion  of  the  scenario  (see 
Appendix  A). 

4.5  Simulation  Data  Recording  Plan 

The  following  data  and  information  was  recorded  during  the  experiment: 


1.  Track  Data 

Track  data  within  the  databases  of  all  the  Horizon  systems  was  logged 
every  30s.  For  each  30s  window,  the  attributes  from  the  last  track  update 
were  used,  unless  the  track  had  not  been  updated  during  this  period,  in 
which  case  the  track  was  not  logged  for  a  second  time.  Dead  reckoned 
tracks  were  not  recorded. 

2.  Ground  Truth 

The  ground  truth  data  from  the  VMS  A  simulation  was  recorded  every  30s. 

3.  Mapping  Files 

Mapping  files  were  recorded  across  all  interfaces  where  tracks  were 
passed  from  one  system  to  another.  These  files  map  the  unique  Track 
Number  used  by  a  local  system  to  the  unique  Track  Number  of  the 
corresponding  track  provided  by  a  remote  system.  These  files  allow  each 
track  within  the  VBE  infrastructure  to  be  mapped  back  to  the 
corresponding  RWO.  This  is  essential  for  the  subsequent  analysis. 

4.  Automated 
Support 

The  time  and  details  of  alerts  and/or  recommendations  from  the 
association  algorithms. 

5.  Video  and 
Photography 

A  photographic  record  and  video  of  the  experiment  was  created. 

6.  Observer  Logs 

The  observers  were  required  to  take  appropriate  manual  logs  of  the 
experiment. 

7.  Horizon  Plugin 
Activation 

Horizon  plugins  activated  by  the  operators  to  assist  in  defining  role  based 
set-ups  for  Horizon  in  future  experiments. 

8.  Display  Screen 
shots 

Screen  snapshots  of  the  Horizon  and  ground  truth  displays  were  taken 
every  10  minutes. 

9.  Workload 

Log  files  from  the  workload  monitoring  application. 

10.  Situation 

Awareness 

SART  survey  instrument 

11.  CO  Comments 

CO  was  provided  with  a  Dictaphone  to  record  comments  on  the  scenario 
and  usability  issues  for  later  correction. 
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4.6  Participant  Briefing 

Prior  to  the  experiment  the  operators  were  assigned  their  roles  and  had  their  responsibilities 
explained.  They  signed  consent  forms.  If  they  were  not  familiar  with  Horizon  they  were  also 
provided  with  informal  training  to  allow  them  to  undertake  their  tasks.  Brief  details  of  the 
scenario  were  also  given  to  provide  the  context  that  would  be  expected  in  normal  operations. 

4.7  Participant  Debriefing 

Following  the  experiment  the  vWaller  command  team  completed  the  SART  (Situational 
Awareness  Rating  Technique)  questionnaire.  SART  was  developed  by  the  Centre  for  Human 
Sciences  at  Farnborough  to  provide  a  subjective  estimation  of  the  perceived  situation 
awareness  of  aircrew.  The  technique  consists  of  rating  items  grouped  in  three  principal 
component  dimensions  of  Understanding,  Supply  and  Demand.  The  Naval  Submarine 
Medical  Research  Laboratory  in  the  US  adapted  the  technique  to  submarine  attack  centre 
tasks  [7]. 


5.  Analysis  of  VBE  AS-4:  Results  and  Discussion 

Our  hypothesis  was: 

IF  track  sharing  occurs  THEN  a  more  complete  and  accurate  representation  of  the  operating 
environment  can  loe  maintained  by  each  platform. 

This  hypothesis  speaks  to  a  central  assumption  that  network  enabling  maritime  forces  will 
deliver  benefits  that  will  enhance  capability.  In  this  study  the  focus  of  capability  enhancement 
was  the  virtual  submarine  platform.  The  approach  taken  here  was  to  address  the  hypothesis 
by  examining  the  comparative  quality  of  the  picture  compiled  by  networked  and  non- 
networked  track  managers. 

5.1  Overview:  Picture  Quality  Assessment 

Figure  11  shows  ground  truth  tracks  mapping  movements  of  various  elements  in  the  scenario. 
Recall  that,  in  this  scenario,  only  vWaller  was  unscripted.  All  other  entities  were  scripted 
constructed  simulations.  vWaller  was  fed  tracks  (simulated  radar  detections)  from  the 
coalition  vessels  (FFH1  and  FFH2)  and  from  the  Unmanned  Aerial  Vehicle  (UAV).  Note  that 
the  origin  of  Figure  11  is  completely  arbitrary.  The  scale  simply  indicates  kilometres  from  that 
origin.  These  ground  truth  tracks  form  the  baseline  from  which  all  subsequent  comparisons 
were  made  during  later  analysis.  The  key  element  we  are  concerned  with  is  vWaller' s 
response  to  scenario  -  at  the  levels  of  both  the  human  and  the  overall  system  response, 
including  developmental  automated  and  part  automated  subsystems. 
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- 1  -  vWaller 

- 2  -  FF  HI 

- 3  -  FFH2 

- 4  -  UAV 

- 5  -  FF  01 

- 6  -  FF  02 

7  -  Merchantl 
“~S  -  Merchant2 

9  -  Merchants 

10  -  Merchant4 

1 1  -  Merchants 

12  -  Merchants 

13  -  Merchant7 

- 14-  Fishl 

- 15-  Fish2 


Figure  11  Scenario:  Ground  Truth  tracks  during  VBE  ASA  Scenario 

5.1.1  Data  analysis  support:  Tantara 

To  assist  with  analysis.  Innovation  Science  was  contracted  to  provide  a  data  processing 
support  tool.  This  was  termed  Tantara.  Based  on  Microsoft's  EXCEL,  Tantara  provided  an 
integrated  set  of  data  logs  and  an  environment  presenting  a  uniform  cluster  of  operations  to 
be  performed  upon  the  VBE  AS-4  data  set. 

5.1.2  An  empirical  analysis  of  comparative  tactical  picture  quality  for  networked 
vs  non-networked  picture  compilation 

As  part  of  the  broader  VBE  process,  an  evolving  set  of  metrics  are  being  collated  and 
documented.  The  concept  of  these  metrics  has  been  based  upon  attributes,  identified  by  the 
US  Single  Integrated  Air  Picture  (SIAP)  project  [9,10],  for  the  assessment  of  picture  quality. 
Details  of  these  metrics  are  to  be  found  in  Manning  &  Arulampalam  [11],  In  this  study  we 
chose  Completeness,  Accuracy  and  Continuity  as  the  basis  of  comparison. 

5.12.1  Completeness 

a.  Detection  Completeness  is  defined  as  the  percentage  of  real  world  objects  detected  during 
each  30  second  capture  of  sensor  data  (In  calculating  this  metric,  ownship  and  coalition 
partners  were  ignored). 
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_  A  ^  1A  Number  of  RWOs  within  picture 

Detection  Completeness  =  - - - xl00% 

Total  number  of  RWOs 

The  metric  can  be  used  to  compare  the  relative  completeness  of  the  Tactical  Picture  compiled 
at  each  Track  Manager  Display.  Since  this  metric  assesses  detection  of  RWOs,  it  was 
appropriate  that  the  Track  Manager  working  display  (where  raw  detections  are  registered) 
rather  than  the  Tactical  Plan  Display  (onto  which  tracks  are  promoted)  be  used. 

Figure  12  shows  the  Detection  Completeness  metric  plotted  at  each  30  second  data  capture 
point  taken  during  the  scenario.  The  figure  gives  a  general  indication  of  the  advantage  of  track 
sharing  for  detection.  The  two  network  capable  pictures  (Net  TM  and  Trials  TM)  registered  a 
greater  percentage  of  possible  detections  overall  than  Ownship  (OSTM)  tactical  picture. 

It  appears  that  the  greatest  advantage  of  track  sharing  arises  early  in  the  scenario  (as 
previously  noted  in  VBE  AS-2).  Some  50-70%  more  detections  were  registered  at  the  Net  TM 
and  Trial  TM  displays.  At  this  level  there  was  little  difference  between  the  Net  TM  and  Trial 
TM  displays.  The  overall  reduction  of  Detection  Completeness  particularly  towards  the  later 
part  of  the  scenario  is  probably  due  to  the  movement  of  a  large  proportion  of  the  contacts 
toward  the  Northwest.  In  the  later  stages  of  the  scenario,  vWaller  was  evidently  able  to  hold  a 
greater  proportion  of  new  contacts  entering  its  sensor  range. 


Time 


Net  TM 
Trial  TM 

OS  TM 


Figure  12  Comparison  of  Detection  Completeness  at  each  Picture  Compilation  process 


b.  Solution  Completeness  is  defined  as  the  percentage  of  real  world  objects  with  position 
solutions.  In  calculating  this  metric,  ownship  and  coalition  partners  were  ignored. 


Solution  Completeness  = 


Number  of  RWOs  within  picture  with  position  solutions 
Total  number  of  RWOs 


xl00% 
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Plotting  this  metric  (see  Figure  13)  reveals  that  the  network  enabled  Track  Managers  (Net  TM 
and  Trials  TM)  were  able  to  generate  solutions  for  between  20%  to  40%  more  of  the  RWOs 
detected  throughout  the  simulation.  The  output  of  the  metric  is  plotted  in  Figure  13.  This 
graph  compares  the  TPD  (Display  onto  which  promoted  tracks  are  displayed)  of  each  track 
manager.  Recall  that  the  TPD  is  the  product  of  track  promotion.  Hence,  it  was  intended  that  it 
include  the  operator's  best  estimate  of  the  position  of  elements  in  the  tactical  picture.  Note 
that  at  about  10:00,  the  picture  compiled  at  the  Net  TPD  appears  to  have  suffered  from  some 
short-term  problem  in  generating  solutions.  It  is  not  clear  at  this  stage  precisely  what  caused 
this  to  arise.  However,  some  of  the  difficulties  faced  by  the  Net  TM  will  be  discussed  in  detail 
later. 


Time 


Figure  13  Solution  Completeness  in  Picture  Compilation:  Tactical  Picture  Display  Comparison 

Comparison  of  networked  and  non-networked  picture  compilation  completion  at  the  Track 
Manager  working  display  is  shown  in  Figure  14.  The  Net  TM  and  Trials  TM  Displays  appear 
to  have  been  quite  similar  in  holding  contacts  with  solutions.  OS  TM  held  between  40  and  70% 
less  vessels  with  solutions. 


Figure  14  Solution  Completeness  in  Picture  Compilation:  Track  Manager  Display  Comparison 
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These  findings  are  not  altogether  surprising  given  the  larger  number  and  distribution  of 
sensors  accessible  by  the  networked  track  management  operators.  In  terms  of  the  tactical 
advantage  of  this  enhanced  picture  completeness,  several  screen  captures  from  the  scenario 
are  revealing.  For  example,  in  Figure  15a  the  OS  TM  display  reveals  only  two  contacts,  while 
the  Net  TM  display.  Figure  15b  shows  considerably  more  contacts  and  in  particular  it  shows 
hostile  vessels  approaching  from  the  South  South-East  of  vWaller.  As  a  final  demonstration  of 
the  extended  field  of  view  of  the  networked  capable  picture,  a  ground  truth  display  is  shown 
in  Figure  15c.  The  central  dotted  circle  signifies  ownship  sensor  ranges  while  networked 
displays  were  capable  of  holding  all  contacts  outside  of  that  small  field  of  view. 


(a)  OS  TM 


(b)  Net  TM 


(c)  Ground  Tmth 


Contacts  not  held 
by  OS  TM 


Figure  15  Comparison  of  Tactical  Pictures  taken  at  10.13am 
5.1.22  Accuracy 

Another  attribute  of  picture  quality  deemed  important  was  the  difference  between  estimated 
position  and  ground  truth  or  Position  Error  (PE).  The  position  error  was  the  difference 
between  the  solution  for  any  given  track  compared  to  the  true  position  of  the  relevant  RWO 
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(clearly,  care  has  to  be  taken  as  to  the  appropriate  use  of  this  metric  for  tracks  that  are  the 
result  of  incorrect  association). 


PE  =  )2  +  (A -A)2 

where  (xk,  yk)  is  the  simulation  coordinate  position  of  an  RWO  and  (xh  yk)  is  the  estimated 
solution  of  the  RWO. 

As  an  initial  description  of  what  this  metric  describes.  Figure  16  compares  the  position  error 
for  both  the  TMA  and  Ownship  solutions  for  Merchant  1.  This  contact  held  constant  speed  on 
a  constant  course  throughout.  The  two  curves  are  intimately  related  since  the  OS  TM  receives 
TMA  output  over  time  as  a  refinement  of  contact  solutions.  Note  that  for  both  curves,  error  is 
initially  quite  large  and  unstable  (between  500  to  5200  meters  compared  to  ground  truth). 
However,  position-error  stabilises  and  is  minimised  during  the  first  25  minutes  of  the 
scenario.  The  initial  error  arises  as  the  TMA  operator  'stacks  the  dots'  on  the  TMA  screen 
(manually  minimises  residual  error  in  localisation). 


3000 


Figure  16  Positional  Error  for  an  example  contact:  Merchant  1  (Ownship  Track  Manager) 


Note  that  the  tracks  generated  at  the  Net  TM  and  Trials  TM  displays  are  derived  from  a 
number  of  different  sensor  sources.  To  demonstrate  the  role  of  each  sensor  in  the  sharing  of 
tracks.  Figure  17  shows  the  contribution  of  each  sensor  as  a  timeline.  Ownship  sonar  held  this 
particular  contact  for  approximately  1/3  of  the  scenario.  Several  passes  of  the  UAV  and  the 
coalition  vessels  held  the  contact  longer. 
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Detection  Source  Timeline:  Merchant  1 

FFH  Radar  (Anzac) 

mm  UAV  Pass  3 

— -  OS  Sonar 

- UAV  pass  2 

^  FFH  Radar  (Arunta) 

— ■  _  UAV  pass  1 

9:30  10:00  10:30  1100  11.30 

Time 


Figure  17  A  timeline  of  sensor  detection  for  an  example  contact:  Merchant  1 

While  Figure  17  shows  the  how  different  sensors  detected  a  particular  vessel  Figure  18 
demonstrates  the  manner  in  which  the  input  of  these  same  sensors  was  managed,  that  is, 
sensor  output  tracks  associated  and  the  impact  of  track  fusion  upon  the  estimated  position  of 
this  Merchant  Vessel  at  any  particular  instant  (in  this  case  estimation  error  has  been  plotted). 

In  Figure  18  each  coloured  line  represents  a  different  sensor  (or  constructed  platform).  Clearly, 
a  major  task  for  the  networked  operators  was  to  manage  constituent  tracks  appropriately,  that 
is,  to  associate  together  the  tracks  emerging  to  optimise  the  solution  (minimise  solution  error). 
For  example,  in  the  case  of  Merchant  1,  Sensors  included  Ownship  (OS)  Sonar,  UAV  radar 
tracks  and  coalition  radar  sensors  (all  'federate'  models).  Separate  fusion  actions  can  be  seen 
where  the  separate  lines  merge.  Four  of  the  major  fusion  actions  involving  this  track  are 
shown.  Note  that  fusion  at  times  actually  increased  error  relative  to  ground  truth  with  a 
subsequent  improvement  in  error.  Some  means  of  filtering  initial  error  generated  by  or 
associations  maybe  useful  -  such  as  a  simple  error-bounding  filter.  The  position  error  shows 
much  about  the  challenges  facing  tracking  in  an  NCW  operational  environment. 
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Figure  18  Subcomponents  of  position  error  for  an  example  contact:  Merchant  1 
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Figure  18  suggests  that  fusion  of  tracks  from  different  sensor  sources  did  not  necessarily 
reduce  the  error  in  localisation.  There  appears  to  have  been  difficulty  fusing  tracks  derived 
from  coalition  sensors  with  ownship  data.  For  example,  the  sector  in  the  error  curve  for 
contact  Merchant  1  increases  dramatically  for  the  constituent  track  that  comprised  Ownship 
detection  of  contact  from  Sonar  3.  Indeed  error  seemed  to  spike  to  peak  at  3,500  m  (at  about 
10:06). 

Given  that  some  interesting  errors  did  arise  in  localisation  and  tracking  of  certain  contacts 
using  track  fusion,  it  is  useful  to  directly  compare  the  average  position  error  for  some  other 
example  tracks  generated  at  each  picture  compilation  operator. 

Figure  19  below  compares  the  average  positional  error  across  all  sensors  for  the  contact 
Merchant  1  at  each  of  the  Track  Manager  Tactical  Picture  Display.  Note  firstly  that  there  is 
substantial  variability  in  errors  generated  at  the  level  of  Ownship  TMA  solutions.  This  error 
reduces  from  about  1500  m  to  below  1000  m  then  to  below  500  m  as  the  Merchant  vessel 
transits  through  sonar  range. 


Figure  19  Average  position  error  for  an  example  contact:  Merchant  1 

In  the  example  of  Merchant  Vessel  2  (see  Figure  20),  a  somewhat  different  pattern  emerges. 
Once  again,  the  overall  value  of  networking  is  clear  from  the  small  proportion  of  time  that 
Ownship  sensors  hold  the  contact.  The  Trials  TPD  also  appears  to  consistently  hold  the  vessel 
with  limited  error  throughout  the  scenario  as  the  Merchant  2  tracks  North  East  past  vWaller's 
position.  The  Net  TPD,  however  appears  to  have  a  problem  with  some  dramatic  error  values. 
A  single  inconsistent  data  point  places  the  average  error  in  position  well  beyond  3,000  m. 
Several  other  spikes  in  error  also  emerge.  This  kind  of  error  is  difficult  to  explain.  Such  error 
could  be  filtered  by  constraining  the  rate  of  change  in  location  of  a  fused  track  according  to 
realistic  motion  rules.  It  appears  to  be  a  possible  risk  for  the  NCW  capability  that 
unpredictable  errors  can  emerge  in  track  management  due  to  perturbation  of  data  aggregation 
tools  such  as  the  fusion  algorithms  used. 
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Figure  20  Components  of  average  position  error  for  an  example  contact:  Merchant  2 

Similarly,  in  the  case  of  priority  contacts,  FFG1  and  FFG2  (hostile  Frigates)  there  are  benefits 
evident  in  the  latency  of  detection  and  tracking  contacts  during  the  scenario  (see  Fig  21  and 
22).  In  Figure  21  the  Net  TPD  plot  of  position  error  is  markedly  more  stable  than  that  for  the 
Trials  TPD.  The  Trials  TPD  held  the  track  for  FFG1  in  excess  of  3000  m  from  its  actual  position 
and  for  a  period  of  over  an  hour.  Even  so,  ownship  TMA  only  tracked  the  vessel  for  a  very 
small  proportion  of  the  actual  scenario  (ownship  TMA).  Once  again  this  points  to  potential 
risks  inherent  in  the  management  of  tracks  made  up  of  multiple  components.  It  is  also 
possible  that  the  level  of  increase  of  information  may  have  led  to  instability  because  the 
solution  space  is  larger.  At  this  stage  it  is  difficult  to  clarify  that  possibility. 


Figure  21  Errors  for  each  networked  picture  for  priority  contact  FFG1  (hostile  Frigate) 

Finally,  in  this  series  of  examples.  Figure  22  plots  the  average  position  error  for  the  other 
hostile  Frigate  -  FFG2.  Once  again,  the  vessel  was  not  detected  by  vWaller  until  the  simulation 
was  almost  over.  Detections  comprising  the  contact  FFG2  until  about  11:55  (when  Ownship 
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sonar  detections  were  initiated)  were  from  offboard  sensors  only.  It  also  appears  that  the 
relative  accuracy  of  Ownship  TMA  can  represent  a  limiting  factor  upon  the  subsequent 
accuracy  of  fused  or  associated  tracks  simply  because  the  error  distance  was  greater  for 
Ownship  sonar  detection. 


Figure  22  Components  of  average  position  error  for  priority  contact  FFG2  (hostile  Frigate) 


The  pattern  of  errors  above  indicates  some  of  the  potentially  difficult  aspects  of  track 
management  in  NCW  operations.  Particularly  when  fusion  is  initiated,  it  can  be  anticipated 
that  errors  will  be  large.  From  our  evidence,  it  takes  time  for  error  to  "normalise". 

These  sources  of  error  should  not  be  over-stated.  In  the  bigger  picture,  things  were  not  as 
unpredictable  as  these  chosen  examples  suggest.  As  the  entire  scenario  unfolded.  Figure  23 
demonstrates  most  RWO  solutions  (i.e.  60  -  90  %)  fell  within  3000m  of  ground  truth.  This 
metric  differs  from  that  above  in  that  it  provides  an  indication  of  the  overall  accuracy  of  all 
tracks  with  solutions.  This  distance  was  nominally  set  to  3km  as  a  typical  search  area  for  a 
torpedo.  In  calculating  this  metric  the  detection  of  own  ship  and  coalition  partners  are 
ignored.  The  metric  used  to  describe  this  characteristic  of  the  situation  is  as  follows,  where 
PROW  is  the  percentage  of  ROWs  with  a  solution  within  x  kilometres  of  its  true  position: 

_ A  T  Number  of  tracks  with  a  positional  error  less  than  x  km  . . .  n , 

PROW  = - - - x  100% 

Total  number  of  detected  tracks  with  solutions 
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Figure  23  RWOs  with  a  solution  that  is  within  3000m  of  ground  truth 

The  range  of  3000m  selected  here  is  probably  reasonable  for  distant  contacts.  However,  when 
contacts  are  high  priority  or  dangerous  then  obviously  the  range  may  not  be  appropriate. 

5.1.3  Continuity  of  Tracking 

Figure  24  below  outlines  the  result  of  applying  the  Detection  Continuity  metric  for  each  track 
in  the  scenario  at  each  Track  Manager. 

Detection  Continuity  (DC)  is  defined  as  the  total  duration  a  given  RWO  (eg.  Merchant  1, 2  etc) 
is  detected,  expressed  as  a  percentage  of  the  total  scenario  duration, 

^  _  Duration  RWO  is  detected  ,  .  , 

DC  = - xl00%. 

Duration  of  scenario 


Clearly,  the  Net  TM  and  Trials  TM  have  held  detection  for  most  contacts  for  longer  duration 
in  this  scenario.  In  particular,  the  priority  contacts  (hostile  FFG1  and  FFG2)  were  held  with 
greater  continuity  when  tracks  were  shared  between  coalition  vessels  (between  30  %  -  60% 
longer).  This  finding  is  scenario  dependent  since  the  hostile  Frigates  approached  the  range  of 
v Wallers's  sensors  only  in  the  last  half-hour  of  the  scenario. 
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Contact 


Figure  24  Comparison  of  track  continuity  between  Track  Managers 


Next,  considering  the  relative  advantage  of  networking  for  track  continuity.  Figure  25  below 
plots  the  continuity  for  each  individual  sensor  platform  in  the  coalition.  This  is  compared  to  a 
plot  of  the  continuity  metric  for  the  entire  coalition.  The  figure  therefore  describes  an 
enhancement  of  tracking  continuity  due  to  sharing  of  sensor  data  within  the  coalition.  The 
network  adds  to  the  overall  continuity  of  tracking. 


Entire  Coalition 


Figure  25  Detection  continuity  across  coalition  sensor  elements  for  each  contact  in  the  scenario 


In  summary  then,  our  evidence  supports  the  first  hypothesis:  that  track  sharing  would 
facilitate  a  more  complete  and  accurate  tactical  picture.  This  was  primarily  because  track 
sharing  essentially  extended  the  effective  sensor  range  of  the  submarine  relative  to  its  onboard 
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sensors.  The  networked  picture  displayed  entities  that  vWaller's  organic  sensors  simply  could 
not  detect.  This  enabled  the  networked  picture  compilation  process  to  access  distant  contacts 
sooner  and  to  track  those  contacts  with  greater  continuity  than  other  was  feasible  for  picture 
compilation  dependent  solely  on  ownship  sensors. 

5.2  Priority  Target  Tracking 

5.2.1  A  second  hypothesis  addressed  was  the  following: 

IF  track  sharing  of  high  priority  targets  occurs  THEN  they  can  be  more  continuously  monitored  with 
greater  accuracy. 

This  second  hypothesis  concerns  high  priority  targets.  In  this  scenario  the  priority  contacts 
were  FFG1  and  FFG2.  These  vessels  were  hostile  contacts.  Figure  26  is  meant  to  summarise  the 
impact  of  track  sharing  within  the  coalition.  The  Figure  plots  the  percentage  enhancement  in 
continuity  achieved  by  networking  platforms  compared  to  ownship  detection  for  each  contact. 

Track  sharing  has  therefore,  on  our  evidence,  enhanced  the  continuity  of  detection  for  the  two 
priority  hostile  FFGs. 


Contact 


Figure  26  Percentage  increase  in  detection  continuity  achieved  by  networking(for  each  contact) 

Although  the  above  figure  demonstrates  the  increase  in  detection  continuity  associated  with 
track  sharing  it  can  not  indicate  whether  or  not  this  has  actually  been  of  any  operational 
benefit.  Determining  this  is  especially  difficult  in  scenarios  that  are  limited  to  general 
surveillance  activities.  The  observation  that  tracks  sourced  from  the  coalition  detected  and 
held  enemy  vessels  (with  low  error)  several  hours  prior  to  ownship  sonar  tracking  does 
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suggest  that  the  Commander  would  have  had  much  longer  to  deal  with  the  enemy  approach3. 
Once  again,  we  cannot  claim  that  this  improves  his  situation  tactically. 

5.3  Area  of  Uncertainty 

Our  third  hypothesis  examined  was  the  following: 

IF  shared  tracks  have  a  realistic  Area  of  Uncertainty  THEN  command  will  have  a  greater  trust  in 
the  information  and  will  associate  and  fuse  them  more  readily  with  own  ship  tracks. 

This  hypothesis  concerns  the  exploratory  development  and  representation  of  an  area  of 
uncertainty  for  the  estimated  positions  of  contacts. 

Area  of  Uncertainty  (AOU)  Accuracy  is  defined  as  the  percentage  of  real  world  objects  lying 
within  the  estimated  target  uncertainty  area  for  their  corresponding  tracks.  The  target 
uncertainty  area  for  a  particular  track  is  determined  by  the  track  estimate,  its  covariance,  and 
a  confidence  level  which  gives  the  probability  that  the  true  object  lies  within  the  uncertainty 
region.  In  calculating  this  metric,  ownship  and  coalition  partners  were  ignored.  To  define  this 
metric,  let 

=  True  position  of  RWO  corresponding  to  track  i 
At  =  AOU  corresponding  to  track  i 
N  =  Number  of  detected  RWO  with  solutions 

and  let  I( TnAt)  be  an  indicator  function  given  by 

SL * 


Then,  the  AOU  Accuracy  can  be  defined  as 


AOU  Accuracy  =  1^1(7;.,  4.)) 


xl00% 


A  great  deal  of  fluctuation  was  evident  in  the  AOU  accuracy  results.  Figure  29  below  is  a  good 
example  of  this.  It  illustrates  the  percentage  of  Real  World  Objects  within  their  actual  AOU  for 
both  the  Net  TPD  and  Trials  TPD  track  data.  The  percentage  of  contacts  that  actually  fell 
within  their  AOU  varied  dramatically  and  this  probably  meant  that  AOUs  were  not  useful, 
that  the  AOUs  were  too  small,  or  that  the  variability  of  positions  was  too  great. 


3  Indeed  the  CO  did  claim  that  the  biggest  advantage  he  saw  in  sharing  tracks  was  that  they 
informed  him  of  the  nature  of  low-bearing  rate  (very  distant)  contacts. 
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a.  Net  TPD 


Figure  27  Percentage  of  vessels  (RWO)  falling  within  calculated  AOU  for  vessels  tracked  in 
Networked  picture  compilation 

Observation  of  the  operators  during  the  simulation  run  saw  no  use  of  the  available  AOU 
display  at  all.  We  cannot  really  comment  on  the  hypothesis  posed  to  address  the  question  of 
trust  in  the  positioning  of  vessels  for  this  scenario. 
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5.4  Human  Performance  Factors 

5.4.1  Workload 

5.4.1.1  Moment  to  moment 

As  a  guide  to  the  workload  to  which  the  operators  were  subjected,  a  simple  moment-to- 
moment  workload  measure  was  carried  out.  This  involved  a  pop-up  screen  containing  a 
rating  scale  (1  =  low  to  7  =  high). 

The  findings  for  this  measure  are  shown  in  Figure  28.  The  figure  suggests  that  the  NetTM 
(Networked  track  manager)  perceived  himself  to  be  under  quite  a  high  workload  in 
comparison  to  the  other  operators.  Both  Ownship  and  Trials  TM  indicated  that  their  workload 
was  very  low.  Given  that  the  Net  TM  and  Trials  TM  tasks  were  very  similar,  one  possible 
explanation  for  the  difference  in  their  perceived  workload  might  be  involvement  of  the  CO 
with  Net  TM  Picture  Compilation.  The  cognitive  effort  of  paying  attention  to  the  CO  in 
discussing  the  picture  layout  may  have  meant  that  perceived  workload  was  high. 


Figure  28  Moment-to-moment  subjective  workload 
5 .4.1.2  Situation  Awareness  Rating  Technique 

The  SART  scores  taken  after  the  simulation  was  run  are  shown  in  Figure  29.  Participants  in 
the  study  appear  to  have  felt  that  they  understood  the  situation  quite  well.  They  also  believed 
they  had  sufficient  resources  with  which  to  deal  with  the  situation.  Interestingly,  the  OS  TM 
whose  focus  was  upon  his  ownship  sensors  believed  his  understanding  of  the  situation  was 
high.  That  is  quite  possible,  however,  relative  to  the  other  TMs,  his  field  of  view  was  small. 

The  degree  that  participants  found  the  tasks  they  performed  to  be  demanding  suggests  that  all 
participants  thought  the  scenario  did  not  challenge  them.  Only  the  Net  TM  rated  Cognitive 
Demand  as  moderate. 
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Figure  29  Situation  Awareness  Rating  Technique  Scores  (Showing  Standard  Error  Bar) 


6.  Concluding  Remarks 


6.1  General  Conclusions 

In  conclusion,  the  simulation  supporting  VBE  AS-4  ran  smoothly.  The  RAN  staff  employed  to 
conduct  picture  compilation  and  to  simulate  control  room  activities  were  largely  engrossed  in 
the  task  at  hand  suggesting  that  efforts  in  immersing  them  in  a  realistic  setting  were  fruitful.  It 
appears  that  the  surveillance  task  faced  by  vWaller  was  a  fair  representation  of  the  type  of 
task  faced  at  sea. 

6.1.1  Tactical  picture  quality  in  a  networked  conventional  submarine 
To  examine  tactical  picture  compilation  three  hypotheses  were  considered: 

1.  IF  track  sharing  occurs  THEN  a  more  complete  and  accurate  representation  of  the 
operating  environment  can  be  maintained  by  each  platform. 

2.  IF  track  sharing  of  high  priority  targets  occurs  THEN  they  can  be  more  continuously 
monitored  with  greater  accuracy. 

3.  If  shared  tracks  have  a  realistic  Area  of  Uncertainty  THEN  command  will  have  a  greater 
trust  in  the  information  and  will  associate  and  fuse  them  more  readily  with  own  ship 
tracks. 

The  evidence  from  this  VBE  study  substantively  supports  hypotheses  1  and  2.  This  is  largely 
due,  it  would  appear,  to  the  increased  'field  of  view'  or  sensor  range  that  is  likely  to  be 
associated  with  NCW  in  the  maritime  domain.  Considerable  picture  quality  benefits  do  seem 
to  have  emerged  for  networked  track  managers.  It  should  be  noted  that  we  have  only 
anecdotal  evidence  that  this  finding  means  that  there  were  tactical  benefits  for  the  CO  of  the 
virtual  submarine.  VBE  AS-4  was  not  designed  to  establish  that  possibility.  Regarding  the 
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third  hypothesis,  we  have  found  that  the  CO  paid  no  attention  to  the  AOU  functionality 
available  and  therefore  we  can  not  support  that  hypothesis.  Given  the  variability  of  the  AOU 
representation  this  is  probably  not  surprising.  A  more  rigorous  exploration  of  the  issue  may 
shed  light  on  the  most  appropriate  means  of  exploiting  visualisation  to  emphasise  the 
uncertainty  inherent  in  track  management. 

6.2  Operator  Performance 

In  this  study  we  did  not  find  a  great  deal  of  difference  between  the  two  networked  operators 
except  in  their  workload.  One  possible  reason  for  this  is  that  the  scenario  was  at  a  tempo  so 
low  as  to  fail  to  generate  the  sort  of  stress  that  one  might  expect  in  a  wargame.  Once  the  work 
of  picture  compilation  becomes  more  hectic  then  there  is  a  good  chance  that  operators  will 
rely  on  the  assistance  of  automation  to  cope. 
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Appendix  A:  SART  Questionnaire 

The  following  questionnaire  forms  the  basis  of  the  Situational  Awareness  Rating  Technique 
(SART)  developed  in  the  UK.  SART  uses  subjective  estimates  of  personal  and  task-dependent 
factors  which  affect  task  performance  and  understanding  to  measure  Situation  Awareness. 
Situation  Awareness  refers  to  your  ability  to  relate  the  meaning  of  events  and  elements  in  a 
noisy,  uncertain  environment  to  mission  goals  and  objectives.  The  technique  involves  the 
scoring  of  fourteen  different  scales,  each  of  which  is  potentially  a  factor  in  your  Situation 
Awareness. 


Each  different  category  should  be  rated  on  a  scale  from  a  low  of  one  to  a  high  of  seven.  These 
values  are  a  subjective  measure  of  your  individual  perceptions  during  the  VBE.  There  is  no 
right  or  wrong  answer  to  give,  only  your  best  estimate  of  your  personal  experience  from  the 
point  of  view  of  your  assigned  role.  Do  not  spend  too  much  time  on  any  one  item.  Your  initial 
'gut  feeling'  is  likely  to  be  the  most  accurate  estimation. 

The  questionnaires  should  be  completed  individually,  but  you  may  ask  for  further 
explanation  of  any  of  the  categories. 

1.  Demand  on  Cognitive  Resources 


How  demanding  was  the  exercise  on  your  cognitive  resources?  Were  there  many  difficult 
decisions  and  situations  demanding  constant  attention  and  maximum  efforts  (high)  or  was  it 
easy  and  minimally  demanding  (low)?  In  other  words,  how  much  did  your  brain  hurt? 


2.  Instability  of  Situations 


How  changeable  were  the  situations  and  environmental  factors  encountered  through  the 
course  of  the  exercise?  Were  they  very  dynamic  and  likely  to  change  suddenly  (high),  or  were 
most  of  them  slow  and  stable  with  easily  predictable  outcomes  (low)? 

P  H  1  (Low)  7  (High) 

Score 


38 


DSTO-TR-1825 


3.  Complexity  of  Situations 


How  complicated  were  the  situations?  Were  they  complex  with  many  interrelated 
components  and  phases  (high),  or  were  most  simple  and  straight  forward  (low)? 


Score 


1  (Low) 


7  (High) 


4.  Variability  of  Situations 


How  many  elements  were  changing  at  any  one  time  in  a  given  situation?  Were  there  a  large 
number  of  dynamic  variables  (high),  or  very  few  that  might  change  at  once  (low)? 


Score 


1  (Low) 


7  (High) 


5.  Supply  of  Cognitive  Resources 


How  great  a  supply  of  cognitive  and  attentional  resources  coupled  with  decision  aids  and 
analysis  tools  did  you  have  for  problem-solving,  decision-making,  and  other  functions  during 
the  exercise?  Could  you  bring  a  very  large  capacity  to  bear  on  the  problems  (high),  or  did  you 
have  limited  resources  at  your  disposal  in  each  situation  (low)?  In  other  words,  did  you  have 
the  assistance  and  resources  you  required  either  from  other  operators  or  computers  to  help 
make  decisions  and  perform  your  job  to  the  required  level? 


Score 


1  (Low) 


7  (High) 


6.  Readiness 


How  alert  and  ready  for  action  did  you  feel  throughout  the  course  of  the  exercise?  Could  you 
anticipate  the  flow  of  events  and  respond  quickly  (high),  or  were  you  hard  pressed  to  keep  up 
with  evolving  situations  (low)? 


Score 


1  (Low) 


7  (High) 


7.  Concentration  of  Attention 


How  much  could  you  concentrate  your  attention  in  each  problem  situation?  Were  you  always 
focused  on  important  elements  and  events  (high),  or  did  technical  details,  controls,  and 
displays  distract  you  and  draw  your  attention  elsewhere  (low)? 


Score 


1  (Low) 


7  (High) 
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8.  Division  of  Attention 


Were  you  able  to  divide  your  attention  among  several  key  issues  in  the  course  of  the  exercise? 
Were  you  usually  concerned  with  many  aspects  of  current  and  future  events  simultaneously 
(high),  or  did  you  focus  on  only  one  thing  at  a  time  (low)? 


Score 


1  (Low) 


7  (High) 


9.  Spare  Mental  Capacity 


How  much  mental  capacity  did  you  have  to  spare  in  this  exercise?  Do  you  think  you  could 
have  dealt  with  a  significant  number  of  additional  elements  and  variables  if  necessary  (high), 
or  did  the  complexity  of  the  exercise  take  all  your  mental  capacity  (low)? 


Score 


1  (Low) 


7  (High) 


10.  Understanding  the  Situation 


How  well  did  you  understand  the  environment,  the  tactical  situations,  the  problems,  and 
tasks  presented  in  the  exercise  just  run?  Include  ownship,  all  contacts,  and  all  sources  of 
information,  as  well  as  mission  goals,  strategy,  and  tactics  for  this  purpose.  In  retrospect,  did 
you  usually  have  a  good  understanding  in  most  cases  (high),  or  did  you  have  many 
unknowns  and  uncertainty  a  major  part  of  the  time  (low)? 


Score 


1  (Low) 


7  (High) 


11.  Information  Quantity 


How  much  useful  information  were  you  able  to  obtain  from  all  available  sources  in  the 
exercise?  Did  you  receive  and  understand  a  great  deal  of  pertinent  data  (high),  or  was  very 
little  of  the  information  of  much  use  for  your  task  at  hand  (low)? 


Score 


1  (Low) 


7  (High) 


12.  Information  Quality 


How  good  was  the  information  you  obtained  about  the  situation?  Was  the  knowledge 
communicated  via  all  sources  very  accurate  and  precise  (high),  or  was  it  noisy  with  high 
levels  of  uncertainty  (low)? 


Score 


1  (Low) 


7  (High) 
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13.  Familiarity  with  the  Environment 


How  familiar  were  you  with  the  different  elements  and  events  in  the  environment  and 
situations  encountered  during  the  course  of  the  exercise?  Could  you  call  on  a  great  deal  of 
relevant  experience  and  knowledge  to  fill  in  gaps  in  the  available  information  (high),  or  did 
you  find  significant  aspects  of  the  exercise  new  and  unfamiliar  to  you  (low)? 


14.  Situation  Awareness 


Evaluate  your  awareness  of  the  overall  meaning  of  events  and  elements  in  the  environment  to 
the  mission  objectives.  Did  you  always  have  a  complete  picture  and  a  plan  for  how  the 
various  elements  would  affect  the  mission  and  could  you  anticipate  future  mission-critical 
events  and  decisions  well  in  advance  (high),  or  did  you  have  very  limited  ability  to  predict  the 
impact  of  on-going  activity  on  future  events  and  overall  mission  goals  (low)? 

~~  ]  1  (Low)  7  (High) 

Score 
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